feat: Event producer graph for graceful shutdown#165
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @chrisk314, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a robust mechanism for graceful shutdown within the system by implementing an event producer graph. This graph allows components to track the status of other components that produce events they consume. The changes also enhance the IOController's shutdown behavior, enabling it to wait for a grace period to ensure all pending events are processed before closing, directly addressing issue #127. This feature improves system stability and reliability during termination.
Highlights
- Event Producer Graph: A new _event_producers attribute is added to the Component class to store a mapping of event types to the IDs of components producing those events. This graph is built during component initialization (_build_producer_graph) and periodically updated (_update_producer_graph) to reflect the current status of event-producing components.
- Graceful Shutdown: The IOController's close method is modified to introduce a IO_CLOSE_GRACE_PERIOD. If there are still input events to process (excluding StopEvent), the controller will wait for this grace period before flushing its internal event buffer, allowing for last events to be processed.
- State Backend Enhancements: New methods (get_process_for_component) are added to both StateBackend and SqliteStateBackend to efficiently retrieve the process data associated with a given component ID, which is crucial for building and updating the producer graph.
- Component IO Information: The Component.dict() method now includes the io information from the IOController, making the component's input/output details available in its dictionary representation.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces an event producer graph to facilitate graceful shutdown of components, which is a solid approach. The changes are well-structured, including periodic updates to the graph and a grace period for event processing upon closing. My review includes a high-severity performance recommendation for fetching component statuses more efficiently, and a couple of medium-severity suggestions to improve code style and readability.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
Benchmark comparison for |
|
Benchmark comparison for |
|
Benchmark comparison for |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Benchmark comparison for |
|
Benchmark comparison for |
|
Benchmark comparison for |
2 participants
Summary
Closes #127 by introducing a component local event producer graph representation which is periodically updated for each component with the current status of other components which produce events of interest. For event driven models, the event producer graph will be updated throughout a model run - possibly only at the end after a long wait without input events. Once there are no more event producers in a component's local graph representation, the component will call
closeon itsIOController, causing it to: process any final received events; and then shutdown in the next IO read cycle.Changes
dumpmethod toIOControllerto track process event topologyComponent.dumpto includeIOControllerdumpget_process_for_componentmethod toStateBackendimplementations to retrieve process event topology within componentsComponentto track state of other event producing componentsIOControllerclose behaviour to wait some grace period for last events on closeComponentIO read behaviour to closeIOControllerwhen no more event producers in case of purely event driven scenario